Video image processing device, video image analysis system, method, and program

ABSTRACT

A video image processing device comprises at least one memory configured to store instructions and, at least one processor configured to execute the instructions. The at least one processor controls to display a trajectory indicating a change in a position of an object in a video image. The at least one processor controls to display an image of the object corresponding to a vicinity of the trajectory in the video image around a part of the trajectory.

The present application is a divisional application of U.S. patentapplication Ser. No. 16/489,374 filed on Aug. 28, 2019, which is aNational Stage Entry of international application PCT/JP2018/006229,filed Feb. 21, 2018, which claims the benefit of priority from JapanesePatent Application 2017-070677 filed on Mar. 31, 2017, the disclosuresof all of which are incorporated in their entirety by reference herein.

TECHNICAL FIELD

The present invention relates to a video image processing device, avideo image processing method, and a video image processing program forprocessing video images. The present invention also relates to a videoimage analysis system for analyzing video images.

BACKGROUND ART

For example, there is a video image analysis technique for analyzing avideo image obtained from a camera device with a computer, and issuingan alert. For example, there is a video image analysis technique fordetecting a position or a motion of an object from a video image, andgenerating an alert when the detection result satisfies a predeterminedcondition. Such a video image analysis technique is utilized at acontrol center where an operator is present, for example, to check avideo image based on which an alert has been issued, and takingappropriate measures in accordance with the issued alert.

In such a case, the operation to check the video image based on whichthe alert has been issued is normally performed while the current videoimage is being monitored at the control center. When an alert is issued,an operator checks the past video images, to determine whether the alertis a false alert. If the alert is not a false alert, the operatoracquires necessary information, and takes measures such as sending theinformation as an appropriate alert to a predetermined address, forexample. In doing so, the operator selects, from the past video images,information from which a check can be made to determine whether thealarm is a false alarm, and, if the alert is not a false alarm, theoperator acquires features of an intruder as the target in the alert, adangerous moving object, or the like (any of these will be hereinafterreferred to as an object).

Regarding such video image monitoring, Patent Literatures 1 to 3disclose example techniques.

For example, Patent Literature 1 discloses that, on the display screenof a display device that displays a moving image, the trajectory ofmovement of an object is superimposed and displayed on imagessequentially obtained from the imaging device that is the source of themoving image. Patent Literature 1 also discloses that designation of themovement trajectory being displayed is received from a user, themovement trajectory is displayed in a different display mode from thatfor the other movement trajectories, and an object detection region isset so that the movement trajectory does not intersect with any othermovement trajectory.

Patent Literature 2 discloses an example in which different persons anda trajectory of one vehicle are shown in a video image.

In Patent Literature 3, displaying a composite image obtained bycombining frame images is disclosed as a method of determining abehavior from displacement of a part. Patent Literature 3 also disclosesan example of combining images by superimposing consecutive frames onone another, and an example of indicating a moving image and a movementtrajectory of a target part with dots and arrows in a composite image.

CITATION LIST Patent Literatures

[Patent Literature 1] Japanese Patent Application Laid-Open No.2015-018340

[Patent Literature 2] Japanese Patent Application Laid-Open No.2009-015827

[Patent Literature 3] Japanese Patent Application Laid-Open No.2012-133666

SUMMARY OF INVENTION Technical Problem

In the fields of surveillance, there is a demand for display thatenables an operator to quickly gather the features of an object frompast images while monitoring the current video image after an alert isissued, and there also is a demand for an interface for the display. Bythe method disclosed in Patent Literature 1, however, a movementtrajectory is always simply superimposed on the current image. While itis possible to observe the movement path of an object, it is notpossible to know the recent and current behaviors of an object shownonly in past video images, the situations in the surrounding area,changes in features, and the like. Therefore, to collect suchinformation, an operator has to switch screens or check past videoimages on another window. This results in the problem of time costs andthe risk of oversight.

The problems with the method disclosed in Patent Literature 2 arebasically the same as those with Patent Literature 1. That is, by themethod disclosed in Patent Literature 2, a movement trajectory is alwayssimply superimposed on the current image. While it is possible toobserve the movement paths of objects, it is not possible to know therecent and current behaviors of an object shown only in past videoimages, the situations in the surrounding area, changes in features, andthe like.

By the method disclosed in Patent Literature 3, consecutive frames aresuperimposed on one another, so that the states of the object or a partof the object in the past can be checked. However, even if this methodis simply applied to a surveillance video image, the image currentlybeing displayed becomes complicated, and it is difficult to obtain thenecessary information.

For example, in a case where consecutive frames in a video image aresimply superimposed on one another, the objects in the respective framesoverlap one another, and the necessary information cannot be obtained.

The problem of time costs and the risk of oversight described abovemight be caused not only in a site of surveillance of a real-time videoimage accompanied by an alert, but also in a scene where the images attwo or more points of time in a video image need to be quickly checked.

The present invention has been made in view of the above problems, andaims to provide a video image processing device, a video image analysissystem, a video image processing method, and a video image processingprogram that enable a user to quickly grasp the situations of an objectat two or more points of time in a video image.

Solution to Problem

A video image processing device according to the present inventioncharacteristically includes a display control means that causes adisplay unit to display a trajectory indicating a change in a positionof an object in a video image, wherein the display control meansacquires a first request designating a point in the trajectory beingdisplayed, and displays a designated image including the object which itwas located at the designated point by superimposing it on an arbitrarybackground image being displayed on the display unit.

A video image analysis system according to the present inventioncharacteristically includes: a tracking means that analyzes a videoimage, and continuously acquires a position of a tracking target objectfrom the video image; a storage means that stores position informationindicating the position acquired by the tracking means in associationwith identification information about an image from which the positionwas acquired in the video image; and a display control means that causesa display unit to display a trajectory indicating a change in theposition of the object in the video image, based on the informationstored in the storage means, wherein the display control means acquiresa first request designating a point in the trajectory being displayed,and displays a designated image including the object which it waslocated at the designated point by superimposing it on an arbitrarybackground image being displayed on the display unit.

A video image processing method according to the present inventioncharacteristically includes: causing a display unit to display atrajectory indicating a change in a position of an object in a videoimage; acquiring a first request designating a point in the trajectorybeing displayed; and displaying a designated image including the objectwhich it was located at the designated point by superimposing it on anarbitrary background image being displayed on the display unit.

A video image processing program according to the present inventioncharacteristically causes a computer to: perform a process of causing adisplay unit to display a trajectory indicating a change in a positionof an object in a video image; in the process, acquire a first requestdesignating a point in the trajectory being displayed; and display adesignated image including the object which it was located at thedesignated point by superimposing it on an arbitrary background imagebeing displayed on the display unit.

Advantageous Effects of Invention

According to the present invention, a user can quickly grasp thesituations of an object at two or more points of time in a video image.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram schematically showing the configuration of a videoimage analysis system of a first embodiment.

FIG. 2 is a block diagram showing an example configuration of the videoimage analysis system of the first embodiment.

FIG. 3 is an explanatory diagram showing an example of detection in atracking unit 103.

FIG. 4 is an explanatory diagram showing an example expression of atrajectory.

FIG. 5 is an explanatory diagram showing an example of correspondencebetween a tracking line and an object.

FIG. 6 is an explanatory diagram showing an example of attachinginformation to a tracking line.

FIG. 7 is an explanatory diagram showing an example of attachinginformation to a tracking line.

FIG. 8 is a flowchart showing an example operation of a video imageanalysis device 2 of the first embodiment.

FIG. 9 is a flowchart showing an example operation of a video imageprocessing device 4 of the first embodiment.

FIG. 10 is a flowchart showing an example of display control in adisplay control unit 105.

FIG. 11 is an explanatory view showing an example of a display image andan example of a superimposed image at the time of the display image.

FIG. 12 is an explanatory view showing an example of a display image andan example of a superimposed image at the time of the display image.

FIG. 13 is an explanatory view showing an example of a display image andan example of a superimposed image at the time of the display image.

FIG. 14 is an explanatory view showing an example of a display image andan example of a superimposed image at the time of the display image.

FIG. 15 is an explanatory view showing an example of a display image andan example of a superimposed image at the time of the display image.

FIG. 16 is an explanatory view showing an example of a display image andan example of a superimposed image at the time of the display image.

FIG. 17 is an explanatory view showing an example of a display image andan example of a superimposed image at the time of the display image.

FIG. 18 is a block diagram showing an example configuration of a videoimage analysis system of a second embodiment.

FIG. 19 is a flowchart showing an example of display control in adisplay control unit 205.

FIG. 20 is a flowchart showing an example of event processing.

FIG. 21 is a flowchart showing an example of event processing.

FIG. 22 is a flowchart showing an example of event processing.

FIG. 23 is a flowchart showing an example of event processing.

FIG. 24 is a flowchart showing an example of event processing.

FIG. 25 is a flowchart showing an example of event processing.

FIG. 26 is an explanatory diagram showing an example of compositepattern of display images.

FIG. 27 is an explanatory view showing an example of a display image andan example of a superimposed image at the time of the display image.

FIG. 28 is an explanatory view showing an example of a display image andan example of a superimposed image at the time of the display image.

FIG. 29 is an explanatory view showing an example of a display image andan example of a superimposed image at the time of the display image.

FIG. 30 is a schematic block diagram showing an example configuration ofa computer according to an embodiment of the present invention.

FIG. 31 is a block diagram showing the outline of a video imageprocessing device of the present invention.

DESCRIPTION OF EMBODIMENTS Exemplary Embodiment 1

The following is a description of embodiments of the present invention,with reference to the drawings. In each of the embodiments describedbelow, an example case where the present invention is applied to videoimage monitoring for fault detection will be described, but the presentinvention is not limited to use in video image monitoring for faultdetection.

FIG. 1 is a diagram schematically showing the configuration of a videoimage analysis system of a first embodiment. As shown in FIG. 1 , thevideo image analysis system 100 includes a video image input device 1, avideo image analysis device 2, a storage device 3, a video imageprocessing device 4, and a display device 5.

The video image input device 1 inputs an image to be analyzed. The videoimage input device 1 is formed with an imaging device capable ofcapturing a moving image, for example. Although only one video imageinput device 1 is shown in FIG. 1 , but more than one video image inputdevice 1 may be used.

In general, a “video image” means a moving image formed with frameimages corresponding to respective consecutive frames. However, in thepresent invention, a “video image” may not be a so-called moving image,but may be a still image group including two or more still imagesaccompanied by information about the imaging times and the imagingregions, a moving image or a composite image group formed with stillimages included in the above described still image group, or the like.Also, in the description below, an image in a video image may not be aso-called frame image in a moving image, but may be an image included inthe above mentioned “video image”. Further, in the description below, animage at a certain time point in a video image may not be the frameimage at the time point in a so-called moving image, but may be theimage of the time corresponding to the time point on a predeterminedtime axis included in the above mentioned “video image”.

The video image analysis device 2 analyzes a video image that has beeninput thereto (this video image will be hereinafter referred to as the“input video image”), stores the analysis result into the storage device3, and outputs a warning or any other message as necessary.

The video image analysis device 2 analyzes the input video image,detects and tracks a predetermined object such as a moving objectappearing in the input video image, and continuously acquires theposition of the object from the input video image. For example, thevideo image analysis device 2 may acquire the position of the object ateach time point in the input video image. The video image analysisdevice 2 also generates analysis information in which positioninformation indicating the acquired position of the object is associatedwith identification information about the image of the time when theposition was acquired. Hereinafter, the object to be tracked will bealso referred to as the “tracking target object” in some cases.

The video image analysis device 2 also analyzes the input video image,and may further detect a correspondence time which is the time point ofthe image in the input video image from which the position of thetracking target object has been acquired, the features of the trackingtarget object in the image in the input video image at the time point,the presence or absence of another object related to the tracking targetobject, and the features thereof if there is such an object.

Examples of the features of an object (which may be the tracking targetobject or another related object) are the states of the object such asthe direction, the size, and operation of the object or a predeterminedpart thereof, changes in the features of the object in the input videoimage, such as the states, the clothes, and the possessions, othermatters related to the object and another related object (such as thepresence/absence of another object, and the classification thereof), andthe like.

The video image analysis device 2 may also output a predeterminedmessage, in accordance with analysis results including those itemsdetected from the input video image. The video image analysis device 2may determine whether the position of the object and the other itemsdetected from the input video image satisfy a predetermined condition,for example. If the video image analysis device 2 determines that thepredetermined condition is satisfied, the video image analysis device 2may output the predetermined message. The video image analysis device 2may output the predetermined message to a predetermined display device,a user terminal, or a predetermined terminal in the control center, forexample.

The storage device 3 stores the input video image and informationindicating the results of the video image analysis conducted by thevideo image analysis device 2. The storage device 3 may store not onlythe input video image but also, for each tracking target object,analysis information that associates information indicating analysisresults including the position acquired from the input video image andother detected items, with information (such as identificationinformation) indicating the image from which the position has beenacquired in the input video image, for example.

It should be noted that the storage device 3 does not necessarily storeall of the input video image. For example, in the case where the inputvideo image is a streaming video that is sequentially input like areal-time video image or where the video image size (data capacity) islarger than a predetermined threshold, the storage device 3 may storeonly a predetermined amount of the video image from the latest part ofthe input video image. Here, a certain number of images, a time length,or a data capacity may be determined as the predetermined amount.

The video image processing device 4 displays, on the display device 5,an image (which may be a moving image and a still image) included at anytime point in the input video image, an image generated in accordancewith the input video image, or the like. The image generated inaccordance with the input video image is not limited to any particularkind. For example, such an image may be a sketch of a predeterminedregion, or may be a composite image formed with images included at twoor more points of time in the input video image. The video imageprocessing device 4 displays an image designated by a user on thedisplay device 5, for example.

The video image processing device 4 has a function of displaying atrajectory indicating changes in the position of the object in the inputvideo image when displaying such an image on the display device 5. Thetrajectory display method will be described later.

The video image processing device 4 may output image information aboutthe image to be displayed on the display device 5, and cause the displaydevice 5 to display the desired image.

The display device 5 is an image display device that displays an imagein accordance with the image information output from the video imageprocessing device 4. The display device 5 is formed with a display orthe like. Although only one display device 5 is shown in FIG. 1 , morethan one display device 5 may be used. In a case where more than onedisplay device 5 is used, the video image processing device 4superimposes the trajectory on the image being displayed on at least onedisplay device 5 designated to display the trajectory by the user.

FIG. 2 is a block diagram showing an example configuration of the videoimage analysis system of this embodiment. It should be noted that FIG. 2shows an example configuration of the video image analysis system on thefunctional aspect. As shown in FIG. 2 , the video image analysis system100 may include a video image input unit 101, a video image holding unit102, a tracking unit 103, an analysis information storage unit 104, adisplay control unit 105, and a display unit 106.

Here, the video image input unit 101 corresponds to the video imageinput device 1. The video image holding unit 102 and the analysisinformation storage unit 104 correspond to the storage device 3. Thetracking unit 103 corresponds to the video image analysis device 2. Thedisplay control unit 105 corresponds to the video image processingdevice 4. The display unit 106 corresponds to the display device 5.

In the example shown in FIG. 2 , the tracking unit 103 is formed with aninformation processing device such as a CPU included in the video imageanalysis device 2, for example. The display control unit 105 is formedwith an information processing device such as a CPU included in thevideo image processing device 4. Although the video image analysisdevice 2 and the video image processing device 4 are shown as separatedevices in FIG. 1 , these devices may be formed with one device.

The video image input unit 101 inputs a video image.

The video image holding unit 102 stores the input video image. It shouldbe noted that the video image holding unit 102 may store only apredetermined amount of the video image from the latest image in theinput video image.

The tracking unit 103 analyzes the input video image, and continuouslyacquires the position of the tracking target object from the input videoimage. The method of tracking the tracking target object with thetracking unit 103 is not limited to any particular method.

When acquiring the position, the tracking unit 103 further detects thetime (the corresponding time) corresponding to the time when theposition was acquired, the features of the tracking target object or thefeatures of another object related to the tracking target object in theimage at the time point in the input video image.

The tracking unit 103 then stores analysis information into the analysisinformation storage unit 104. In the analysis information, informationindicating analysis results including the position acquired from theinput video image and the items detected together with the position isassociated with identification information about the image from whichthe position was acquired in the input video image. Here, theidentification information is not limited to any particular information,as long as it can identify the image in the input video image. Forexample, the identification information may be information indicating atime point of the image in the input video image, or an identifierattached to the image in the video image.

Further, the tracking unit 103 may have a function of outputting apredetermined message in accordance with the analysis results or otherinformation.

The tracking unit 103 may output a message indicating an alert, inaccordance with sensor information that is input from a predeterminedsensor such as an infrared sensor, a pressure sensor, or a vibrationsensor. In a case where an object is detected from the input video image(an object other than the background is seen in the video image) as aresult of analysis of the input video image, the tracking unit 103 mayoutput a message to that effect. In a case where a predetermined object(such as a human being or a specific person Y) is detected from theinput video image as a result of analysis of the input video image, thetracking unit 103 may output a message to that effect. In a case where amoving object crossing over a predetermined intrusion detection line isdetected, or where a moving object entering a predetermined region or anobject being left behind in or taken away from the predetermined region,for example, the tracking unit 103 may output a message to that effect.

FIG. 3 items (A) to (D) are explanatory diagrams showing examples ofdetection in the tracking unit 103. The example shown in item (A) ofFIG. 3 is an example in which an object (a moving object) is detectedfrom an image region a01 of an input video image. It should be notedthat sign T represents the object regarded as the tracking targetobject. The example shown in item (B) of FIG. 3 is an example in which apredetermined object is detected from the image region a01 of the inputvideo image. It should be noted that sign a02 represents an objectregion that is the region of the object detected from the image in theinput video image. The example shown in item (C) of FIG. 3 is an examplein which an object crossing over an intrusion detection line a03 in theimage region a01 of the input video image is detected. Further, theexample shown in item (D) of FIG. 3 is an example in which an objectbeing left behind in a predetermined monitoring region a04 of the imageregion a01 of the input video image is detected. Here, the object beingleft behind is detected as a related object ro that is another objectrelated to the tracking target object T.

The tracking unit 103 may store the information indicating the positionacquired as a result of tracking separately from the other information.For each tracking target object, the tracking unit 103 may store, intothe analysis information storage unit 104, information that associatesthe information indicating the position with the identificationinformation about the image in the input video image, separately frominformation that associates the information indicating the detecteditems other than the position with the identification information aboutthe image in the input video image. In this embodiment, a combination ofthese kinds of information even in such a case is referred to as“analysis information”.

The display control unit 105 causes the display unit 106 to display atrajectory indicating changes in the position of the object in the videoimage. For example, when displaying an image at some time point includedin the input video image or a predetermined image generated inaccordance with the input video image, the display control unit 105 setsthe image as the background image, and superimposes the trajectory ofthe predetermined object in the input video image on the backgroundimage. It should be noted that the background image is not limited toany particular kind of image, as long as it includes a regioncorresponding to at least a part of the moving path of the object in theimage region.

Further, in accordance with the information stored in the analysisinformation storage unit 104, the display control unit 105 of thisembodiment adds a notification function of notifying the user of theresults of the analysis of the input video image and the time elapsedsince a predetermined time, to the trajectory in the input video imageof the object.

For example, the display control unit 105 may make the display mode of apart of the trajectory differ from another part, or add informationindicating the analysis results or the elapsed time in the vicinity of apart of the trajectory, in accordance with the features of the object orthe features of another object related to the object shown in theanalysis result obtained by analyzing the input video image.Alternatively, the display control unit 105 may make the display mode ofa part of the trajectory differ from another part, or add informationindicating the elapsed time in the vicinity of a part of the trajectory,in accordance with the elapsed time from the predetermined time, forexample.

The analysis results are not limited to any particular results, as longas they are items obtained as a result of analyzing the input videoimage. The analysis results may be the object-related items obtained asa result of tracking the object shown in the input video image (theitems may be the features of the object or the features of anotherobject related to the object), for example. Examples of such itemsinclude the state of the object, changes in the features of the object,the presence or absence of another object related to the object, andinformation amount such a related object, as described above. The stateof the object may be the direction, the size, movement (motions orbehaviors), and the like of the entire object or a predetermined part ofthe object.

An example of another object related to the object is an object thatdraws a trajectory having a predetermined relationship with thetrajectory of the object in the input video image. It should be notedthat the above described “being left behind” and “being take away”, andthe later described “interaction” are examples of the predeterminedrelationship.

Also, the corresponding position in the trajectory accompanied byinformation may be the position in the trajectory corresponding to thetime point when an item to be displayed was detected, for example. In acase where the time elapsed since the predetermined time is to bedisplayed, the corresponding position in the trajectory is the positionin the trajectory corresponding to the time when the elapsed time passedsince the predetermined time

The display control unit 105 may set the “vicinity” that is apredetermined pixel range from the corresponding position in thetrajectory, for example.

The predetermined time may be the latest time in the input video imageor the time when the predetermined message is sent from thepredetermined system that analyzes the input video image.

Further, in a case where a point or a section designated by a user or apoint or a section satisfying a predetermined condition as a result ofanalysis, is included in the trajectory, the display control unit 105may display the trajectory in such a form that the point or the sectioncan be recognized, when superimposing the trajectory.

For example, the display control unit 105 may make the display mode of apart (such as the point or the section) of the trajectory differ fromanother part, or add information indicating the analysis results or theelapsed time in the vicinity of a part of the trajectory (such the pointor the section).

Hereinafter, such a point that satisfies a condition may be referred toas a condition point. Likewise, such a section satisfying a conditionmay be referred to as a condition section.

The condition point or the condition section may be a point or a sectionwhere the direction, the size, or movement of the object or apredetermined part thereof satisfies a predetermined condition, or apoint or a section where changes in the features of the object satisfy apredetermined condition, for example.

Further, in a case where the trajectory includes a roaming section inwhich the object is determined to be roaming according to the analysisresults, for example, the display control unit 105 may shorten thesubject section, and then display the section in a different displaymode from the other sections. It should be noted that the roamingsection is also an example of a condition section.

The display control unit 105 may determine whether a section in thetrajectory is a roaming section, in accordance with the roughness ratioof a trajectory that is the trajectory of sections of respective timeunites in the trajectory of the object or the trajectory of one or moresections that follows the time unit sections and has a predeterminedthickness, to the extensional rectangle surrounding the trajectory. Themethod of determining a roaming section is not limited to the abovedescribed method. For example, a roaming section may be determineddepending on whether the current position is located in a region basedon the past position within the set time or whether the change in theposition at each set time is equal to or smaller than a predetermineddistance.

For example, in a case where a roaming section is included in thetrajectory, and the roaming section is shortened for display, thedisplay control unit 105 acquires a predetermined request designatingthe shortened and displayed section. The display control unit 105 maythen restore the section to the original state and display the originalsection, or display an alternative display including the pointscorresponding to the points included in the original section in thedesignated section.

The display control unit 105 may display a line or a slide bar as theabove mentioned alternative display. The line or the slide barcorresponds to the section prior to the shortening in the display regionof the display unit 106 as the above alternative display. At this stage,it is preferable to make the line segment the same display target as thesubject section or attach the same information to the line segment.

Further, in a case where the input video image is a streaming videoimage to be sequentially input, for example, the display control unit105 may set one of an image at some time point in the input video imageand a predetermined image generated based on the input video image as abackground image, and display a trajectory superimposed on thebackground image. With this arrangement, the user can constantlyrecognize the latest elapsed time and the state of the object.

At this stage, even if the background image is an image at a past timepoint included in the input video image, for example, the displaycontrol unit 105 can superimpose, on the background image, a trajectoryin the display mode corresponding to the results of the latest analysisof the object detected from the input video image, or a trajectory inthe display mode corresponding to the latest time elapsed since thepredetermined time. With this arrangement, while checking the backgroundimage as a past image, the user can recognize the latest situation bychecking a trajectory including the latest position of the objectdisplayed together with the image, the latest features and the like ofthe object, and a trajectory that has a display mode or accompanyinginformation varying with the elapsed time.

In this embodiment, a trajectory that has a display mode or accompanyinginformation varying with the above described analysis results or thetime elapsed since the predetermined time is also called a trackingline.

For example, the display control unit 105 can display the followinginformation (features), using the tracking line.

The traveling direction of the object (which is the tracking line)

The time elapsed since the predetermined time

The direction of the face of the object

The staying time at the corresponding point (using display of theelapsed time and display of the above point or the section)

Features such as the color or the pattern of the outfit of the object

A motion of the object (such as crouching, standing up, walking, running(speed), or jumping)

An interaction with another object (such as crossing, joining, orbranching)

To whom the tracking line belongs

Examples of interactions with another object include an interactionbetween persons and an interaction with another object (such as avehicle). Here, the presence or absence of an interaction may bedetermined depending on whether there is a point or section located in arange close to the trajectory of another object in terms of time anddistance in the trajectory of the object. In a case where the trajectoryof the other object moves away after an interaction like a conversationbetween persons, the interaction may be displayed as “crossing”. In acase where two or more trajectories are combined into one by aninteraction like persons riding a vehicle, the interaction may bedisplayed as “joining”. In a case where two or more trajectories arederived from one trajectory by an interaction like persons getting off avehicle, the interaction may be displayed as “branching”.

Using the object whose trajectory is to be displayed and using thetrajectory, the display control unit 105 may narrow down object(s) todisplay the trajectory, display item(s) to be displayed on the displayunit 106 using the trajectory, and range of the trajectory to bedisplayed, based on user's designation, a reference time or a referencetiming for the elapsed time, the time elapsed from the reference, adirection of the object, staying time of the object, a color of anoutfit of the object, a pattern of an outfit of the object, and aprogressing direction of the object, for example.

The display control unit 105 can also display the trajectories of two ormore objects in one background image. For example, the display controlunit 105 may display an interaction with another object related to theobject by displaying the trajectory of the object.

The display mode in a trajectory can be varied by changing colors, linetypes (including the shapes of dotted lines and the like), spacing(including spacing between the dots and spacing between lines in dottedlines), line thicknesses, the shapes of elements such as the partscorresponding to the lines in dotted lines or markers, or the directionsof the markers or directional signs, for example. It should be notedthat these items can be used in combination.

Here, the markers may be marks indicating the spacing between the spotsor the sections included in a line segment, or marks, figures, or someother signs that represent regular items that always exist in thetrajectory. Symbols are any appropriate characters, marks, figures, andother signs that are provided only when a specific condition issatisfied. Alternatively, the display control unit 105 may vary displaymodes in a trajectory by changing the display method (narrowed display)when narrowing down predetermined sections in the trajectory.

FIGS. 4(a) to 4(j) are explanatory diagrams each showing an exampleexpression of a trajectory (tracking line). FIG. 4(a) shows an examplein which color density is varied in trajectory. FIG. 4(b) shows anexample in which color is varied in a trajectory. In FIG. 4(b),differences in color are represented by shades of the line. FIG. 4(c)shows an example in which marker intervals are varied in a trajectory.FIG. 4(d) shows an example in which the shapes of markers are varied ina trajectory. FIG. 4(e) shows an example in which the directions ofdirectional signs attached in a trajectory are varied. The directionalsigns are associated with directions of the object, for example.Although arrows are shown as an example of the directional signs in FIG.4(e), the directional signs may be simple lines without any marks attheir tops, or may have some other shapes such as triangular shapes.FIG. 4(f) shows an example in which symbols indicating specificinformation are attached to the corresponding points. FIG. 4(g) shows anexample in which display modes are varied as specific markers are madeto flicker.

FIG. 4(h) shows an example of narrowed display of sections, and anexample in which the range other than a specific range is grayed out.FIG. 4(i) shows another example of narrowed display of sections, and anexample in which the range other than a specific range (such as asection within a specific time period) is erased. FIG. 4(j) showsanother example of narrowed display of sections, and an example in whichthe thickness of the tracking line is varied between the current sectionand the other sections. For narrowed display, it is preferable to varydisplay modes by using a different method from the method used to varydisplay modes in a specific range.

FIG. 5 is an explanatory view showing an example of correspondencebetween a tracking line and an object. As shown in FIG. 5 , the displaycontrol unit 105 may associate a tracking line with an object byassigning the same number or the like to the tracking line and theobject. In the drawing, sign TL represents the tracking line. Thetracking line TL in the drawing is colored in accordance with the stateof the object. However, in FIG. 5 , the different colors are indicatedby shades of the line.

In a case where one image includes more than one tracking target person,and images of a certain object at different points of time are displayedat the same time, the display control unit 105 may indicate the identityof the object by assigning the same number or the like to the vicinitiesof object regions of the object, or surrounding the object regions ofthe object with frames in the same color or lines of the same type.

FIGS. 6 and 7 are explanatory diagrams showing examples in whichinformation is attached to a tracking line. FIG. 6 shows an example ofaccompanying information to be attached to a trajectory in a case whereimages of a certain object at different points of time in the trajectoryare simultaneously displayed. It should be noted that images of theobject at the other points of time other than the background image arealso included in the accompanying information.

For example, FIG. 6 shows an example in which the line type of the framesurrounding the object region a02 of the object is varied, and symbolsare added to the frames on display. In the example shown in FIG. 6 , theframes of object regions a02-1 to a02-3 are indicated by solid lines inaccordance with a change in the features of the object (the presence orabsence of a coat), and the frame of an object region a02-4 Is indicatedby a dot-and-dash line.

In this manner, the line types of frames may be varied depending on thestate of the object. FIG. 6 also shows an example in which an imageobtained by cutting out the object region of the object included in theimage at the time point corresponding to the time when an alert wasissued (an alerting time point) in the input video image is furthersuperimposed, and a symbol (the circled A in the drawing) to that effectis attached to the frame line of the object region a02-2. The circled Rin the drawing is an example of a symbol attached at the time pointcorresponding to the current time, and the circle P in the drawing is anexample of a symbol attached to the time point corresponding to a pasttime. In this manner, the “vicinity” of a certain time point in thetrajectory also includes the vicinity of the object region a02 of theobject displayed in accordance with the point. The information to beattached also includes a clipped image (hereinafter referred to as anobject image) of the object region of the object included in the imagegenerated when the object is located at the point within the trajectory.

Also, the exclamation mark in the drawing is an example of a symbolindicating that a feature has changed. The “ro” mark in the drawing isan example of a symbol indicating that there is an interaction withanother object. In a case where the object is not accompanied with anyobject image, similar symbols may be attached near the correspondingpoints on the tracking line.

Further, as shown in FIG. 7 , the display control unit 105 may attachthe information about the object at the time point corresponding to acertain point such as a point designated by the user in the trajectory,or the information about the elapsed time at the time pointcorresponding to the certain point. Other than that, the display controlunit 105 may express the corresponding time with color or accompanyinginformation, by varying the colors of the frame lines at the alertingtime point, some other past time point, and the current time point, forexample.

Further, the display control unit 105 may allow the user to designatethe target to display the trajectory, the display items, and the rangeof the trajectory, from the elapsed time and the analysis results. Forexample, a graphical user interface (GUI) capable of designating theitems listed below may be prepared so that the user can narrow down theobject to display the trajectory, the display items, the range of thetrajectory, and the like.

Examples of Targets to Be Narrowed Down.

-   -   The time or the timing set as the reference time for the elapsed        time, and the time elapsed since the reference time (Within        minutes from the time of alert issuance, for example)    -   The direction of the object    -   The Staying time of the object    -   The color or the pattern of the outfit of the object    -   The traveling direction of the object

The GUI may be a general menu expression such as a combo box, a list, acheck box, a radio button, a text input, or time selection.

Next, an operation according to this embodiment is described. FIG. 8 isa flowchart showing an example operation of the video image analysisdevice 2 (the tracking unit 103) of this embodiment. In the exampleshown in FIG. 8 , a video image to be analyzed is first input from thevideo image input device 1 (step S11).

The video image analysis device 2 then tracks the tracking target objectin the input video image, continuously acquires the position of thetracking target object, and detects a predetermined item about thetracking target object in the image of the corresponding time or of thetime when the position was acquired (step S12: a video image analysisprocess).

The video image analysis device 2 then outputs the video imageaccompanied by the analysis result (step S13: a video image output withan analysis result). Instead of the above output, the video imageanalysis device 2 may associate the input video image with informationindicating the analysis result, and store the input video image and theinformation into the storage device 3.

The video image analysis device 2 repeats the above processing in stepsS11 to S13 until the video image input is ended (step S14).

FIG. 9 is a flowchart showing an example operation of the video imageprocessing device 4 (the display control unit 105) of this embodiment.In the example shown in FIG. 9 , a video image accompanied by ananalysis result is first input from the video image analysis device 2 orthe like (step S21). The video image processing device 4 may read theinput video image and the information indicating the analysis resultfrom a predetermined storage unit, instead of receiving an input of avideo image accompanied by an analysis result.

When the video image is input, the video image processing device 4displays, on the display device 5, the image at a certain time point inthe input video image or an image created in accordance with the inputvideo image (step S22: a display control process). The display controlprocess will be described later in detail.

The video image processing device 4 repeats the above processing insteps S21 and S22 until an end of display is detected (step S23).

In the description below, an example of the display control process instep S22 is explained, with the functional block being the main body ofthe operation. FIG. 10 is a flowchart showing an example of the displaycontrol (the above display control process in step S22) in the displaycontrol unit 105.

In this example, it is assumed that, prior to step S101, the backgroundimage and the object to be displayed are designated by the user or aredetermined in advance. It is also assumed that an image that can be abackground image (such as a predetermined amount of image in the inputvideo image or an image generated from the input video image) is storedin the video image holding unit 102 together with its identifier.Further, it is assumed that the analysis information storage unit 104stores analysis information in which information indicating the analysisresults including the position acquired from the input video image bythe tracking unit 103 and the other detected items is associated withidentification information about the image from which the position wasacquired in the input video image.

In the example shown in FIG. 10 , the display control unit 105 firstacquires a background image from the video image holding unit 102 (stepS101).

The display control unit 105 then acquires the analysis information fromthe analysis information storage unit 104 (step S102).

The display control unit 105 then generates the object's trajectorysuitable for the background image, in accordance with region informationabout the background image (step S103). Here, the region information isinformation that associates the coordinates of the background image withthe coordinates of the (real) imaging region. In the trajectorygeneration, a trajectory image in which only the trajectory (trackingline) to be displayed is drawn in the image region corresponding to thebackground image should be generated. The technique of calculating ordrawing the path of a trajectory line suitable for a known backgroundimage whose positional relationship between the region in the image andthe imaging region in accordance with the position informationindicating the continuous positions of the object is a known technique,and therefore, detailed explanation thereof is not made herein.

In step S103, in accordance with the analysis information, the displaycontrol unit 105 generates a trajectory accompanied by informationattached in the vicinity of the corresponding position in thetrajectory. The information indicates whether the display mode in thetrajectory varies with the results of the analysis of the input videoimage or the time elapsed since the predetermined time, or indicates theresults of the analysis of the input video image or the elapsed time.

The display control unit 105 then superimposes the generated trajectoryon the background image, to generate a display image that is an imagefor display (step S104).

Lastly, the display control unit 105 outputs the image data of thegenerated display image to the display unit 106, and causes the displayunit 106 to display the display image (step S105).

Next, examples of display images in this embodiment are described.

FIG. 11 to FIG. 17 are explanatory views showing examples of displayimages. In each of these drawings, (A) is an explanatory view showing animage obtained by converting a display image as a color image into asimplified image, and (B) is an explanatory view showing a simplifiedsuperimposed image that is an image other than the background image inthe display image. It should be noted that, as the display image issimplified, some parts showed using different colors originally isshowed differently by using another method (such as attaching differentkinds of lines or marks).

The example shown in FIG. 11 is an example of a display image in a casewhere the latest image is set as the background image, and thebackground image including the latest object is updated every moment. InFIG. 11 , the image of the object corresponding to the latest time is animage of the background image. In item (A) of FIG. 11 the display mode(specifically, the color) of the trajectory is varied with the timeelapsed since the latest time, while a symbol (such as circled A,circled P, circled R, or “ro” mark) corresponding to the state of theobject at the corresponding time is provided.

In item (B) of FIG. 11 , because of the limitations of the drawing,differences in the elapsed time are expressed with line types (spacingbetween the lines in dotted lines), instead of colors. As shown in FIG.11 , the display control unit 105 may superimpose and displayinformation other than the tracking line, such as the intrusiondetection line a03.

The example shown in FIG. 12 is an example of a display image in a casewhere the image at the alerting time point is set as the backgroundimage, and only the tracking line is updated every moment. As shown inFIG. 12 , the display control unit 105 may superimpose a trajectorybased on the latest position of the object while displaying the latestimage as well as a past image. As a result, it becomes possible toobserve another tracking target object (another object related to anintruder, for example). In the example shown in item (A) of FIG. 12 thelines surrounding the object regions of the object and another objectare different, being a solid line and a dashed line, and the colors ofthe lines are varied for each object, for example. In item (B) of FIG.12 , instead of colors, numbers for identifying objects are provided.

In the example shown in FIG. 12 , the latest object image is notincluded. However, the display control unit 105 may further performpredetermined processing, such as transparency boosting, on the latestobject image (an image cut out from an object region of the object) orthe entire latest image, and then superimpose the processed image on thebackground image.

The example shown in FIG. 13 is an example of a display image in a casewhere a tracking target object other than a tracking target object foran alert, such as an intruder, is designated as the object. As shown inFIG. 13 , the object whose trajectory is to be displayed is not limitedto any particular object, and may be any tracking target objectdesignated by the user, for example.

The example shown in FIG. 14 is an example of a display image in a casewhere there is another object related to the object. As shown in FIG. 14, in a case where a connection is recognized between the trajectories ofobjects such as when a person gets off a vehicle, the display controlunit 105 may determine that these objects are related to each other, andsuperimpose and display the trajectory of the other object, as well asthe trajectory of the object, on the background image.

In the example case shown in FIG. 14 , the latest image is set as thebackground image, and the background image including the latest objectis updated every moment. However, the background image at the time whenthe trajectory of a related object is displayed, and the method ofupdating the background image are not limited to any particular imageand any particular method.

Further, in a case where an interaction between the object and anotherobject is detected, the display control unit 105 may display thetrajectory of the other object having the interaction with the object,together with the trajectory of the object, even though the interactionis not the object getting off a vehicle. In such a case, a similarnotification function may be given to the trajectory of other object.

The example shown in FIG. 15 is an example of a display image on which atrajectory including a roaming section is superimposed. In FIG. 15 ,items (A-1) and (B-1) are an example of a display image and an exampleof a simplified superimposed image on which a roaming section issuperimposed without shortening, respectively. Reference numeral all initem (B-1) indicates the region in which the object roaming has beendetected. Also, items (A-2) and (B-2) are an example of a display imageand an example of a simplified superimposed image on which the roamingsection is superimposed after being shortened. Further, items (A-3) and(B-3) are an example of a display image and an example of a simplifiedsuperimposed image on which the roaming section is superimposed afterbeing shortened and expanded. For example, in a case where the displaycontrol unit 105 determines that the object is roaming around aparticular region, the display control unit 105 may set a roamingsection that is the sections in the trajectory corresponding to theroaming part, and turn the analysis results within the section into agroup, to display only typical information.

When performing the grouping, the display control unit 105 preferablydisplays the information in a mode indicating that the section is agrouped and shortened section (by attaching some symbol or providing anarrowed display, for example). In this example, items (A-1) and (B-1)correspond to the display image prior to the grouping (in a normalstate), and items (A-2) and (B-2) correspond to the display image afterthe grouping.

In a case where the display control unit 105 receives from the user anexpansion instruction designating the section in the display image afterthe grouping, the display control unit 105 may expand and display thesection, as shown in items (A-3) and (B-3). In this case, an instructioninput about the point within the section after the expansion can bereceived through a direct click on the mouse wheel or the tracking lineafter the expansion, an operation of a slide bar or the likealternatively displayed in a predetermined region on the display screen,or the like.

The example shown in FIG. 16 is an example in which a section in aspecific range designated by the user is displayed as an annotationsection in such a mode that the annotation section can be recognized. Itshould be noted that the display control unit 105 may add timeinformation and image information corresponding to the annotationsection to the analysis information so that the supervisor who is notpresent can be notified of the annotation section. Hereinafter, theinformation to be added in accordance with such an annotation sectionwill also be referred to as annotation information.

Further, the display control unit 105 may cut out the video image (imagegroup) corresponding to the annotation section, and output the videoimage to a predetermined device. As a result, the checking costs of thesupervisor can be lowered.

The example shown in FIG. 17 is an example in which the results ofclassification of changes in the direction and the posture of the objectare expressed with line types. As shown in FIG. 17 , where the displaymodes are varied within a trajectory in accordance with changes in thestate and the features of the object, the operator can determine ofwhich time point an image should be checked to examine the appearance ofthe object.

As described above, in this embodiment, the trajectory of the object isdisplayed on the screen currently being checked, while the display modeof a part of the trajectory is made to differ from another part inaccordance with the time elapsed since the predetermined time and theresults of the analysis of the input video image, or informationassociated with a point in the trajectory is attached. Thus, the usercan intuitively know the elapsed time and the analysis results inassociation not only with the traveling direction of the object but alsowith the point in the trajectory of the object.

For example, from the trajectory of the object, the user can know theposition of the object (or where the object was located) of the timewhen a certain period of time elapsed since the predetermined time, andhow the object spent time before and after the certain period of timeelapsed. From the trajectory of the object, the user can also recognizethe situation of the time of intrusion (whether the intrusion reallyoccurred), the state of the object (classification results such as thedirection, the posture, the motion, and the color of the outfit), andthe presence or absence of another object related to the object.

That is, the display mode of a part of a trajectory is changed inaccordance with the time elapsed since the predetermined time or theanalysis results (the direction and movement of the object, forexample), or information is attached to the vicinity of a part of thetrajectory, so that it becomes easy to select at which point in thetrajectory the image should be checked to observe the details of theobject. Specifically, unless the display mode of a part of thetrajectory is changed or information is attached to the vicinity of apart of the trajectory, for example, it is not possible to determine atwhich point in the trajectory the object should be checked to observethe details of the object. As a result, it takes a long time to checkthe details of the object. In this embodiment, on the other hand, thetrajectory of the point corresponding to the time point when the objectis facing toward the camera is thicker than the other points. As aresult, in a case where the details of the object are checked from pastvideo images while the current video image is being checked, the thickpart of the trajectory should be selected, and the object at the timepoint corresponding to the point, or the object located at the point,should be checked. Thus, the details of the object can be easilychecked. It should be noted that the above effect can also be achievedby narrowing down the objects whose trajectories are to be displayed,the display items, or the range of trajectories, in accordance withdesignation from the user or a predetermined condition.

In the above described manner, it is possible to present to the user thetime elapsed since the predetermined time and the results of analysis ofthe object by using a trajectory. Thus, the user can promptly recognizethe situations of the object at two or more points of time in a videoimage, in accordance with the information presented by the trajectorycurrently being displayed. For example, while checking the currentimage, the user can quickly determine from what point in the past animage should be selected to obtain desired information. The user canalso recognize the current position of the object while checking a pastimage, for example.

Also, depending on the information to be attached to a trajectory, theimage of the object at a specific time point (such as an alert issuancetime) is superimposed and displayed in the vicinity of the correspondingpoint in the trajectory, so that the user can know the situations of theobject at two or more points of time without switching screens.

As will be described later in detail in a second embodiment, the usermay be enabled to designate from which time point the image to besuperimposed should be selected, using a trajectory. In this manner, theuser can acquire necessary information, without complicating the screendisplay. Even when such an instruction is received from the user, thetrajectory display according to this embodiment can aid the user indetermining from which time point the object should be selected andchecked. Thus, the user can promptly grasp the situations of the objectat two or more points of time in a video image.

Exemplary Embodiment 2

Next, a second embodiment of the present invention is described. In thisembodiment, the display control unit 105 has a GUI function in atrajectory superimposed and displayed on a background image. Morespecifically, the video image processing device 4 further has a GUIfunction of acquiring a request (predetermined instruction input) thatdesignates a point in the trajectory being displayed, the request beingassociated with the trajectory superimposed and displayed on thebackground image, and performing screen control in accordance with therequest.

With this function, it becomes possible for the user to check the objectof the time when the object was located at the designated point in thesame background image along the trajectory, only by tracing thetrajectory currently being displayed. Further, the user is enabled tosimultaneously display the object from two or more points of time (suchas the time point of the background image and the time pointcorresponding to a designated point, or the time point corresponding toa first designated point and the time point corresponding to a seconddesignated point), and to switch background images.

In the description below, the same components as those of the firstembodiment are denoted by the same reference numerals as those used inthe first embodiment, and explanation of them will not be made.

The system configuration of a video image analysis system of thisembodiment is basically the same as the configuration of the firstembodiment shown in FIG. 1 . In the example described below, the videoimage analysis device 2 (the tracking unit 103) detects the position ofa tracking target object, and also detects information other than theposition (such as the corresponding time, the features of the trackingtarget object, and another object related to the tracking targetobject). However, it is not necessary to detect the information. Thatis, the video image analysis device 2 (the tracking unit 103) of thisembodiment should be able to analyze an input video image, and generateanalysis information in which position information indicating theposition of the tracking target object in the input video image isassociated with identification information about the image from whichthe position was acquired.

FIG. 18 is a block diagram showing an example configuration of the videoimage analysis system of this embodiment. The video image analysissystem 200 shown in FIG. 18 includes a display control unit 205, insteadof the display control unit 105 of the video image analysis system 100of the first embodiment shown in FIG. 2 .

The display control unit 205 displays the trajectory of a predeterminedobject in an input video image on the display unit 106. For example,when displaying an input video image, an image at some time pointincluded in the input video image, or a predetermined image generated inaccordance with the input video image, the display control unit 205 setsthe image as the background image, and superimposes the trajectory ofthe predetermined object in the input video image on the backgroundimage. In this embodiment, the background image is not limited to anyparticular image, as long as it is an image including the regioncorresponding to at least a part of the moving path of the object in theimage region.

The display control unit 205 of this embodiment further adds a GUIfunction to the trajectory of the object in the input video image. TheGUI function is for acquiring a predetermined request that designates apoint in the trajectory currently being displayed, and performingdisplay control in accordance with the request.

For example, the display control unit 205 acquires a first requestdesignating a point in the trajectory currently being displayed, anddisplays a designated image superimposed on an appropriate backgroundimage being displayed on the display unit 106. The designated imageincludes the object of the time when the object was located at thedesignated point. Hereinafter, the first request will be also referredto as a “designated image addition request”. The point designated by thepredetermined request including the first request will be also referredto as the “designated point”. In a case where a section is designated inthe request, the section will be also referred to as the “designatedsection”.

At this stage, the display control unit 205 may superimpose a designatedimage, every time the designated point changes. The designated imageincludes the object of the time when the object was located at thedesignated point. In this manner, the user can be enabled to check theobject shown in the image corresponding to the time point of a pointedspot in the same image, simply by moving the position of the pointedspot in the trajectory. It should be noted that the display control unit205 may superimpose a designated image that is an image generated whenthe object was located at the designated point among the images includedin the video image (this image will be hereinafter referred to as thecorresponding image).

For example, in a case where a first request is a request that is inputtogether with the position information about a pointed spot when thespot pointed by a pointing device is moved in a trajectory, the displaycontrol unit 205 may superimpose, every time a first request isacquired, a designated image including the object of the time when theobject was located at the point indicated by the position information asthe designated point. In this manner, the user can check the object ofthe time when the object was located at the designated point in the sameimage, simply by tracing the trajectory.

For example, in a case where an image is obtained by cutting out theobject region of an object from an image (corresponding image) generatedwhen the object was located at a designated point among the imagesincluded in a video image, and the obtained image is used as thedesignated image, the display control unit 205 superimposes thedesignated image in the position corresponding to the position of theobject in the corresponding image in the background image. The displaycontrol unit 205 may superimpose the designated image on the backgroundimage after determining the position and the size in which thedesignated image is superimposed on the background image in accordancewith the position and the size of the object in the corresponding image,for example.

Further, in a case where an image obtained by cutting out the objectfrom the corresponding image is used as the designated image, forexample, the display control unit 205 may superimpose a seconddesignated image, together with the designated image, on the backgroundimage. The second designated image is obtained by cutting out anotherobject related to the object from the corresponding image.

The display control unit 205 may also acquire a second requestdesignating a point in the trajectory currently being displayed, andswitch the background image to the corresponding image. At the same timeas the switching, the display control unit 205 may superimpose anddisplay the trajectory of the object on the switched background image.Hereinafter, the second request will also be referred to as the“background switching request”.

The display control unit 205 may also acquire a third requestdesignating a point or a section in the trajectory currently beingdisplayed. The display control unit 205 may then add informationindicating the designation to the image (corresponding image) generatedwhen the object was located at the designated point or section among theimages included in the video image, or may extract the image and outputthe image to the outside. Further, the display control unit 205 may setthe section at this stage as the annotation section. The display controlunit 205 may then make the display mode of the section differ from theother sections in the image currently being displayed, and addinformation indicating the variation to the section. Hereinafter, athird request will also be referred to as an “annotation additionrequest”.

When displaying a trajectory, the display control unit 205 may alsosuperimpose a designated image, together with the trajectory, on thebackground image. The designated image is obtained from the image of atime when a predetermined condition was satisfied in the input videoimage. This is equivalent to the display of a trajectory accompanied bythe object image of a specific point time in the first embodiment.

When superimposing the designated image on the background image, thedisplay control unit 205 may further superimpose identificationinformation about the object or information about the time pointcorresponding to a designated point, on the designated image, forexample. This is equivalent to the associating by assigning the samenumbers or the like to indicate the identity of the object, and theadding of information (such as circled P, circled A, and circled R) inaccordance with the elapsed time or the corresponding time in the firstembodiment.

The display control unit 205 may also acquire a fourth requestdesignating a point or a section in the trajectory currently beingdisplayed, for example. The display control unit 205 may then display adesignated image on the background image until an instruction to cancelthe fourth request is acquired. The designated image includes the objectof the time when the object was located at the designated point.Hereinafter, the fourth request will also be referred to as the“designated image pinning request”.

Without acquiring any request, the display control unit 205 canconstantly display the image of the object of an alerting time point orthe image of a related object at the alerting time point, for example,after adding a symbol or the like indicating the alerting time point, asin the first embodiment.

Further, the display control unit 205 may add a GUI function not only toa trajectory but also to the object region of the tracking target objectincluded in a display image. That is, in a case where tracking targetobjects other than the current object are displayed in a display image(the tracking target objects are shown in or superimposed on thebackground image), the display control unit 205 can cause the user toselect the object whose trajectory is to be newly displayed from amongthe tracking target objects. For example, in a situation where trackingtarget objects other than the current object are displayed on the screenas shown in FIG. 12 , when an object switching request that designates atracking target object other than the current object is acquired fromthe user, the display control unit 205 may set the designated trackingtarget object as the new object, and display the trajectory of the newobject on the current background image. At this stage, the trajectory ofthe previous object may be left behind or erased, or may be selected bythe user.

Further, the display control unit 205 can also perform control todisplay time information and the detected item that triggered an alert,receive a user operation or the like in response to the display, anddisplay the image of the time corresponding to those items. In thiscase, the display method may be a method of further superimposing theimage on the display image currently being displayed, or a method ofdisplaying the image by switching background images. In a case where theimage is further superimposed, the entire image of the correspondingtime may be superimposed, or a part (the corresponding part) of theimage may be cut out and then be superimposed.

In this embodiment, the display control unit 205 is also formed with aninformation processing device such as a CPU included in the video imageprocessing device 4.

Next, an operation according to this embodiment is described. FIG. 19 isa flowchart showing an example of display control in the display controlunit 205.

In this example, it is also assumed that, prior to step S201, thebackground image and the object to be displayed are designated by theuser or are determined in advance. It is also assumed that an image thatcan be a background image (such as a predetermined amount of image in aninput video image or an image generated from the input video image) isstored in the video image holding unit 102 together with its identifier.Further, it is assumed that the analysis information storage unit 104stores analysis information in which information indicating the analysisresults including the position acquired from the input video image bythe tracking unit 103 and the other detected items is associated withidentification information about the image from which the position wasacquired in the input video image.

In the example described below, the image information about the displayimage to be displayed on the display unit 106 is divided into layers,and the layers are stored and managed. The layers are then superimposedon one another, and are output. However, the method of generating adisplay image is not limited to this example.

In the example shown in FIG. 19 , the display control unit 205 firstacquires a background image from the video image holding unit 102, andsets the background image in a background layer (step S201). Morespecifically, setting in a layer means storing the image informationabout the image to be displayed in the layer into a buffer provided forthe layer.

The display control unit 205 then acquires analysis information from theanalysis information storage unit 104 (step S202).

The display control unit 205 then generates a trajectory of the objectsuitable for the background image in accordance with the regioninformation about the background image, and sets the generatedtrajectory in a trajectory layer (steps S203 and S204). At this stage,the display control unit 205 generates a trajectory (a trajectory image)in which a point is time in the trajectory is associated with the imageor the time point in the input video image or time information about thetime point.

The display control unit 205 then superimposes the image information inthe background layer and the image information in the trajectory layeron each other, and stores the superimposed image information into thedisplay buffer that stores the image information to be output to thedisplay unit 106 (step S205).

At this stage, if there is an object image or the like to be displayedin advance, the display control unit 205 may convert the object image asa designated image into an image in the position and the sizecorresponding to those of the corresponding image set as the backgroundimage. The display control unit 205 may then set the object image,together with a pinning flag, in a superimposed image layer in which thedesignated image is to be set. The number of superimposed image layersis equal to the number of images to be superimposed on one another.

The display control unit 205 then determines whether an image is set ina superimposed image layer (step S206), to superimpose the designatedimages set so far on the background image. If an image is set in asuperimposed image layer (Yes in step S206), the display control unit205 further superimposes and stores the set image into the displaybuffer (step S207).

If images are set in two or more superimposed image layers, the displaycontrol unit 205 superimposes and stores all the set images into thedisplay buffer in step S207. The display control unit 205 then moves onto step S208.

If any image is not set in the superimposed image layers (No in stepS206), on the other hand, the display control unit 205 moves on directlyto step S208.

In step S208, the display control unit 205 outputs the image informationstored in the display buffer to the display unit 106.

Through the above described operation in steps S201 to S208, a displayimage in which the background image, the trajectory, and, if any, theimage(s) set in the superimposed image layers are superimposed on oneanother is displayed on the display unit 106.

In the situation where an image is displayed on the display unit 106,the display control unit 205 acquires a predetermined request includingdesignation of a point in the trajectory currently being displayed. Inthis example, the display control unit 205 receives an eventcorresponding to the request. The display control unit 205 then performsthe processing corresponding to the event (step S209: event processing).Examples of the event processing are shown in FIGS. 20 to 25 .

The display control unit 205 may return to step S202 after apredetermined time has passed since the end of the event processing, forexample, and acquire a request while repeating the operation in stepsS202 to S208, to update the trajectory.

Referring now to FIGS. 20 to 25 , examples of the event processing aredescribed.

FIG. 20 is a flowchart showing an example of the event processing inaccordance with a first request (designated image addition request). Inthe example shown in FIG. 20 , the display control unit 205 performs theprocessing in steps E11 to E14 in a case where the accepted event is afirst request event indicating that a first request has been acquired(Yes in step E11).

The display control unit 205 first clears the superimposed imagelayer(s) to which the pinning flag is not attached (step E12). Here, thepinning flag is the flag indicating that the image in the correspondingsuperimposed image layer is to be constantly displayed. In step E12, thedesignated point image(s) set in the superimposed image layer(s) towhich the pinning flag, which was displayed before the first request,can be cleared at the next display image update timing.

The display control unit 205 then acquires the designated image from thecorresponding image corresponding to the point designated by the firstrequest related to the event, adjusts the position and the size asnecessary, and sets the designated image in a new superimposed imagelayer (steps E13 and E14). To reflect the setting contents in thedisplay image, the display control unit 205 returns to step S206.

In this manner, the designated image obtained from the correspondingimage corresponding to the designated point is superimposed on thecurrent display image.

FIG. 21 is a flowchart showing an example of event processingcorresponding to a first request cancellation event indicating that afirst request is invalidated. Here, the first request cancellation eventsupposedly occurs when the point designated by a first request moves toanother position or when a first request indicating a new point isreceived.

In the example shown in FIG. 21 , the display control unit 205 performsthe processing in step E16 in a case where the accepted event is a firstrequest cancellation event (Yes in step E15).

Specifically, the display control unit 205 clears the superimposed imagelayer(s) to which the pinning flag is not attached (step E16).

As a result, at the display screen update timing, the designated imagethat is superimposed on the current display image and corresponds to thepoint prior to the movement or the point designated by the previousfirst request is cleared. It should be noted that, after step E16, thedisplay control unit 205 may immediately return to step S205 and updatethe display screen.

FIG. 22 is a flowchart showing an example of event processingcorresponding to a second request (a background switching request). Inthe example shown in FIG. 22 , the display control unit 205 performs theprocessing in steps E22 and E23 in a case where the accepted event is asecond request event indicating that a second request has been acquired(Yes in step E21).

The display control unit 205 first clears all of the trajectory layerand the superimposed image layers (step E22).

The display control unit 205 then sets a background image that is thecorresponding image corresponding to the designated point (step E23). Toreflect the setting contents in the display image, the display controlunit 205 returns to step S201.

As a result, the corresponding image is set as the background image, anda display image in which a trajectory is superimposed on the backgroundimage is displayed on the display unit 106. At this stage, if there isan object image to be displayed, the display control unit 205 mayconvert the object image as a designated image into an image in theposition and the size corresponding to those of the corresponding imageset as the background image. The display control unit 205 may then setthe object image, together with a pinning flag, in a superimposed layer.

FIG. 23 is a flowchart showing an example of event processingcorresponding to a third request (an annotation addition request). Inthe example shown in FIG. 23 , the display control unit 205 performs theprocessing in step E32 in a case where the accepted event is a thirdrequest event indicating that a third request has been acquired (Yes instep E31).

Specifically, the display control unit 205 adds annotation information(information indicating that the user has issued an instruction) to theimage corresponding to the designated point or the designated sectionand the trajectory (step E32).

It should be noted that the display control unit 205 may further cut outan image corresponding to the designated point or the designated sectionfrom an input video image, and output the image to the outside.

FIG. 24 is a flowchart showing an example of event processingcorresponding to a fourth request (a designated image pinning request).In the example shown in FIG. 24 , the display control unit 205 performsthe processing in step E42 in a case where the accepted event is afourth request event indicating that a fourth request has been acquired(Yes in step E41).

Specifically, the display control unit 205 sets a pinning flag in thesuperimposed image layer in which the designated image corresponding tothe designated point is set (step E42).

In this manner, the designated image currently being displayed isprevented from being cleared at the next display screen update timingand later.

FIG. 25 is a flowchart showing an example of event processingcorresponding to a fifth request event corresponding to a fifth request(a fourth request cancellation request). In the example shown in FIG. 25, the display control unit 205 performs the processing in step E52 in acase where the accepted event is a fifth request event indicating that afifth request has been acquired (Yes in step E51).

Specifically, the display control unit 205 cancels the pinning flag inthe superimposed image layer in which the designated image correspondingto the designated point is set (step E52).

In this manner, the designated image currently being displayed in apinned manner is cleared at the next display screen update timing.

Next, examples of display images in this embodiment are described.First, examples of composite patterns of display images in thisembodiment are described.

FIG. 26 is an explanatory diagram showing an example of generationpatterns for composite images (display images) to be generated in thisembodiment. In FIG. 26 , examples of generation patterns for displayimages from which trajectories are removed are shown. As shown in FIG.26 , the display control unit 205 may generate a display image bysuperimposing an image cut out from a designated point image on a pastimage, for example (generation pattern 1). Here, the designated pointimage corresponds to the above mentioned corresponding image. The imagecut out from the designated point image corresponds to the abovementioned designated image.

Also, the display control unit 205 may generate a display image bysuperimposing an image cut out from the latest image and an image cutout from a designated point image on a past image, for example(generation pattern 2). Here, the image cut out from the latest imagemay be the object image of the object included in the latest image.

Also, the display control unit 205 may generate a display image bysuperimposing an image cut out from a designated point image on thelatest image, for example (generation pattern 3).

Further, the display control unit 205 may generate a display image bysuperimposing an image cut out from a past image and an image cut outfrom a designated point image on the latest image, for example(generation pattern 4). Here, the image cut out from a past image may bean object image of an object or a related object included in any of thepast images. Examples of such object images include an object image ofan object or a related object included in the past image at a specifictime point, such as an alerting time point, a time point when a featurechange was detected, or a time point when there was an interaction withanother related object.

FIGS. 27 to 29 are explanatory views showing examples of display imagesaccording to this embodiment. In each of these drawings in thisembodiment, item (A) is an explanatory view showing an image obtained byconverting a display image as a color image into a gray scale image, anditem (B) is an explanatory view showing a simplified superimposed imagethat is an image other than the background image in the display image,as in the first embodiment.

The example shown in FIG. 27 is an example of the display image to bedisplayed after the user issues a first request designating the pointcorresponding to a certain past time point in a trajectory in a displayimage in which the latest image is set as the background image and thebackground image including the latest object is updated every moment. Inthis example, the point designated by the first request is indicated bya white arrow.

As shown in FIG. 27 , in response to the first request, the displaycontrol unit 205 may further superimpose and display a designated image(the object image at that time) on the display image. The designatedimage is formed by cutting out an object region a02-3 of the object fromthe corresponding image corresponding to the designated point. In thisexample, before the first request is acquired, a trajectory and adesignated image obtained by cutting out an object region a02-2 of theobject from the corresponding image at the time of issuance of an alertare superimposed on the latest image.

The example shown in FIG. 28 is an example of a display image that isdisplayed while the designated image to be superimposed on the displayimage is changed with movement of such a first request. In the drawing,the object region a02-3, an object region a02-4, and an object regiona02-5 are displayed at the same time. In practice, however, theseregions are switched on display, in accordance with movement of thepointed spot (the white arrow in the drawing).

The example shown in FIG. 28 is an example of a display device in a casewhere the user designates a certain time point in the past in asituation where the latest image is set as the background image, and thebackground image including the latest object is updated every moment. Asshown in FIG. 28 , in a case where the user designates a point in thetracking line of the object, for example, the display control unit 105may superimpose and display the object image (the object region a02-3 inthe drawing) of the object of the time when the object was located atthe point, and an object image of a related object (see the objectregion a02-4 in the drawing). In this manner, the user may also beenabled to check the characteristics and the like of a related object.

In a case where an interaction between the object and another object,other than the object getting off a vehicle, is detected, the displaycontrol unit 205 may set the image of the other object of the time ofthe interaction as an image of a related object, and superimpose anddisplay the image, together with the image of the object of the sametime point.

As described above, according to this embodiment, the user simplydesignates a trajectory currently being displayed, an object region ofthe object accompanying the trajectory, or an object region of a relatedobject, and makes a predetermined input. In this manner, a cut-out imagecorresponding to a designated point can be displayed, or backgroundimages can be switched. Thus, the states of the object at two or morepoints of time in a video image and the states of the surroundings canbe promptly grasped.

In the above described example, the display control unit 205 of thisembodiment also adds a notification function to a trajectory asdescribed in the first embodiment. However, the display control unit 205can add only a GUI function to a trajectory, without giving such anotification function to the trajectory. That is, it is possible toprovide a GUI function of this embodiment even for simple trajectorydisplay.

Next, an example configuration of a computer according to an embodimentof the present invention is described. FIG. 30 is a schematic blockdiagram showing an example configuration of a computer according to anembodiment of the present invention. A computer 1000 includes a CPU1001, a main storage device 1002, an auxiliary storage device 1003, aninterface 1004, a display device 1005, and an input device 1006.

The video image analysis device and the video image processing devicedescribed above may be mounted on the computer 1000, for example. Inthat case, operations of the respective devices may be stored as aprogram in the auxiliary storage device 1003. The CPU 1001 reads theprogram from the auxiliary storage device 1003, loads the program intothe main storage device 1002, and performs predetermined processingaccording to the above embodiments, in accordance with the program.

The auxiliary storage device 1003 is an example of a non-transitoryphysical medium. Other examples of non-transitory physical media includemagnetic disks, magneto-optical disks, CD-ROMs, DVD-ROMs, semiconductormemories, and the like to be connected to the computer 1000 via theinterface 1004. Further, in a case where this program is delivered tothe computer 1000 via a communication line, the computer 1000 may loadthe program into the main storage device 1002 after receiving thedelivery, and perform predetermined processing according to the aboveembodiments.

Further, the program may be for performing part of the predeterminedprocessing in each embodiment. Furthermore, the program may be adifferential program for performing predetermined processing accordingto the above embodiments in combination with another program alreadystored in the auxiliary storage device 1003.

The interface 1004 transmits and receives information to and from otherdevices. The display device 1005 presents information to users. Theinput device 1006 receives inputs of information from users.

Depending on the details of processing in an embodiment, some of thecomponents of the computer 1000 can be omitted. For example, if thedevice does not present information to users, the display device 1005can be omitted.

Part or all of each component of each device is implemented bygeneral-purpose or special circuitry, processors or the like, orcombinations thereof. These may be formed with a single chip or may beformed with chips connected via a bus. Alternatively, part or all ofeach component of each device may be formed with a combination of theabove mentioned circuitry or the like and a program.

In the case where part or all of each component of each device is formedwith information processing devices and circuitry or the like, theinformation processing devices and the circuitry or the like may bearranged in a centralized manner or a distributed manner. For example,the information processing devices and the circuitry or the like may beformed with a client and server system, a cloud computing system, andthe like connected to one another via a communication network.

FIG. 31 is a block diagram showing the outline of a video imageprocessing device of the present invention. As shown in FIG. 31 , thevideo image processing device 50 of the present invention may include adisplay control means 501.

The display control means 501 (the display control unit 105 or 205, forexample) acquires a first request designating a point in the trajectorybeing displayed, and displays a designated image including the objectwhich it was located at the designated point by superimposing it on anarbitrary background image being displayed on the display unit.

Instead of the above processing, according to a feature of the objectshown in an analysis result obtained as a result of analysis of a videoimage, a feature of another object related to the object shown in theanalysis results, or an time elapsed time from a predetermined time, thedisplay control means 501 may make the display mode of a part of thetrajectory differ from another part, or attach information indicatingthe analysis result or the elapsed time to the vicinity of a part of thetrajectory.

By adding a specific notification function and a GUI function to thetrajectory, the user can promptly grasp the situations of the object attwo or more points of time in a video image.

The exemplary embodiments disclosed above can be described as thefollowing supplementary notes.

Supplementary Note 1.

A video image processing device comprising a display control means thatcauses a display unit to display a trajectory indicating a change in aposition of an object in a video image, wherein the display controlmeans acquires a first request designating a point in the trajectorybeing displayed, and displays a designated image including the objectwhich it was located at the designated point by superimposing it on anarbitrary background image being displayed on the display unit.

Supplementary Note 2.

The video image processing device according to Supplementary note 1,wherein, every time the designated point changes, the display controlmeans superimposes, on the background image, the designated imageincluding the object which it was located at the designated point.

Supplementary Note 3.

The video image processing device according to Supplementary note 1 or2, wherein the designated image is an image obtained by cutting out theobject from a corresponding image being an image generated when theobject was located at the designated point among images included in thevideo image and, when superimposing the designated image on thebackground image, the display control means superimposes a seconddesignated image together with the designated image on the backgroundimage, the second designated image being an image obtained by cuttingout another object related to the object from the corresponding image.

Supplementary Note 4.

The video image processing device according to Supplementary note 3,wherein the another object related to the object draws a trajectoryhaving a predetermined relationship with the trajectory of the object inthe video image.

Supplementary Note 5.

The video image processing device according to any of Supplementarynotes 1 to 4, wherein the display control means acquires a secondrequest designating a point in the trajectory being displayed, andswitches the background image to a corresponding image being an imagegenerated when the object was located at the designated point amongimages included in the video image.

Supplementary Note 6.

The video image processing device according to any of Supplementarynotes 1 to 5, wherein the display control means acquires a third requestdesignating one of a point and a section in the trajectory beingdisplayed, and attaches information indicating the designation to animage generated when the object was located at the designated one of thepoint and the section among images included in the video image, orextracts the image from the video image and outputs the image tooutside.

Supplementary Note 7.

The video image processing device according to any of Supplementarynotes 1 to 6, wherein the display control means attaches identificationinformation about the object or information about time corresponding tothe designated point, to the designated image, and superimposes thedesignated image on the background image.

Supplementary Note 8.

The video image processing device according to any of Supplementarynotes 1 to 7, wherein the display control means, in accordance with oneof a feature of the object shown in an analysis result obtained byanalyzing the video image, a feature of another object related to theobject shown in the analysis result, and an elapsed time from apredetermined time, makes a display mode of a part of the trajectorydiffer from another part or attach information indicating one of theanalysis result and the elapsed time to a vicinity of a part of thetrajectory.

Supplementary Note 9.

A video image analysis system comprising:

a tracking means that analyzes a video image, and continuously acquiresa position of a tracking target object from the video image; a storagemeans that stores position information indicating the position acquiredby the tracking means in association with identification informationabout an image from which the position was acquired in the video image;and a display control means that causes a display unit to display atrajectory indicating a change in the position of the object in thevideo image, based on the information stored in the storage means,wherein the display control means acquires a first request designating apoint in the trajectory being displayed, and displays a designated imageincluding the object which it was located at the designated point bysuperimposing it on an arbitrary background image being displayed on thedisplay unit.

Supplementary Note 10.

A video image processing method comprising:

causing a display unit to display a trajectory indicating a change in aposition of an object in a video image; acquiring a first requestdesignating a point in the trajectory being displayed; and displaying adesignated image including the object which it was located at thedesignated point by superimposing it on an arbitrary background imagebeing displayed on the display unit.

Supplementary Note 11.

A video image processing program for causing a computer to:

perform a process of causing a display unit to display a trajectoryindicating a change in a position of an object in a video image; in theprocess, acquire a first request designating a point in the trajectorybeing displayed; and display a designated image including the objectwhich it was located at the designated point by superimposing it on anarbitrary background image being displayed on the display unit.

Although the present invention has been described with reference to thisembodiment and examples, the present invention is not limited to theabove embodiments and examples. Various changes that can be understoodby those skilled in the art within the scope of the present inventioncan be made to the configuration and details of the present invention.

This application claims priority based on Japanese Patent Application2017-070667, filed Mar. 31, 2017, the entire disclosure of which isincorporated herein.

INDUSTRIAL APPLICABILITY

The present invention can be suitably used not only in surveillance, butalso in checking a video image accompanied by analysis information. Forexample, the present invention can be suitably used in marketing torecognize the situations of customers from a video image taken in astore or in the vicinity of a specific item.

REFERENCE SIGNS LIST

-   100, 200: Video image analysis system-   1: Video image input device-   2: Video image analysis device-   3: Storage device-   4: Video image processing device-   5: Display device-   101: Video image input unit-   102: Video image holding unit-   103: Tracking unit-   104: Analysis information storage unit-   105, 205: Display control unit-   106: Display unit-   1000: Computer-   1001: CPU-   1002: Main storage device-   1003: Auxiliary storage device-   1004: Interface-   1005: Display device-   1006: Input device-   50: Video image processing device-   501: Display control means

The invention claimed is:
 1. A video image analyzing system comprising:at least one memory configured to store instructions; and at least oneprocessor configured to execute the instructions to perform: analyzing avideo image, and detecting a tracking target object from the videoimage; displaying a background image which is a composite image formedwith images included in the video image at two or more points of time,and an image including an object region of the tracking target objectincluded in another image generated when the tracking target object islocated, on a display device, displaying a trajectory indicating changesin position of the tracking target object, and when the trajectoryincludes a roaming section in which the tracking target object isdetermined to be roaming, shortening the roaming section and displaysthe roaming section in a different display mode from other displayedsections.
 2. The video image analyzing system according to claim 1,wherein the at least one processor displays information about time whenthe tracking target object is located.
 3. The video image analyzingsystem according to claim 1, wherein the at least one processor receivesdesignation of the tracking target object from a user.
 4. The videoimage analyzing system according to claim 1, wherein the at least oneprocessor displays the image including the object region in the vicinityof a trajectory of the tracking target object.
 5. The video imageanalyzing system according to claim 1, wherein the background imagecorresponds to at least a part of a moving path of the tracking targetobject.
 6. The video image analyzing system according to claim 1,wherein the at least one processor displays an image representing a fullbody of a person as the image including the object region.
 7. A videoimage analyzing method comprising: analyzing a video image, anddetecting a tracking target object from the video image; displaying abackground image which is a composite image formed with images includedin the video image at two or more points of time, and an image includingan object region of the tracking target object included in another imagegenerated when the tracking target object is located, on a displaydevice; displaying a trajectory indicating changes in position of thetracking target object; and when the trajectory includes a roamingsection in which the tracking target object is determined to be roaming,shortening the roaming section and displaying the roaming section in adifferent display mode from other displayed sections.
 8. Anon-transitory computer-readable recording medium in which a video imageanalyzing program is recorded, wherein the video image analyzing programcauses a computer to perform: a process of analyzing a video image, anddetecting a tracking target object from the video image; a process ofdisplaying a background image which is a composite image formed withimages included in the video image at two or more points of time, and animage including an object region of the tracking target object includedin another image generated when the tracking target object is located,on a display device; a process of displaying a trajectory indicatingchanges in position of the tracking target object; and a process of,when the trajectory includes a roaming section in which the trackingtarget object is determined to be roaming, shortening the roamingsection and displaying the roaming section in a different display modefrom other displayed sections.